acceptodds
Under review as a conference paper at ICLR 2027

Improving Chain-of-Thought Monitorability in Large Language Models: Causal Bounds and Post-Training

Abstract

Chain-of-thought (CoT) monitoring aims to detect potentially harmful behaviors in language models from their reasoning traces. Intervention-based evaluations measure this property by asking whether a monitor can detect the role of an intervention from the CoT when the targeted behavior occurs, commonly using minimal-criterion sensitivity () guan2025monitoring. However, a target outcome observed under intervention may also have occurred without it. For stable binary intervention experiments with a positive average effect, we show that upper-bounds the monitor's recall on intervention-caused outcomes and derive a corresponding lower bound. We then study whether post-training improves measured CoT monitorability under a fixed monitoring setup and whether the observed gains persist when sensitivity is replaced by this lower bound. We construct a training dataset of paired control–intervention tasks spanning contextual evidence, answer cues, sandbagging, and deception, and use it to train models with supervised fine-tuning, reinforcement learning with monitoring-based rewards, and on-policy self-distillation. Across most evaluated tasks, all three methods improve measured monitorability relative to the base model under both the standard composite and its causal-recall lower-bound counterpart.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Chain-of-thoughts, monitorability, supervised fine-tuning, reinforcement learning, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RNLKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper derives a new lower bound on causal recall and shows that the existing minimal-criterion sensitivity serves as its upper bound. It also investigates how to improve cot monitorability by constructing a synthetic training dataset with paired control and intervention inputs across different task categories. The authors explore three post-training approaches: sft on the constructed dataset, rl with a monitorability-oriented reward design, and opsd where a teacher with privileged information guides the student using differences in token distributions. Experimental results show improvements in measured monitorability under both the standard metric and the causal-recall lower-bound metric, with sft showing the most consistent improvements.

Critical strengths

The authors present (1) a more rigorous theoretical foundation for evaluating CoT monitorability and clarify the causal interpretation of existing metrics.

(2) A comprehensive experimental evaluation using well-designed synthetic datasets and multiple post-training paradigms, with some novelty in the reward design and OPSD framework.

(3) Positive results showing improvements in CoT monitorability.

Critical weaknesses

(1) I don't understand why the authors presented three different training frameworks. What distinct insights do they provide beyond showing that post-training can improve monitorability?

(2) The analysis is not sufficient, despite the dataset is diverse, monitorability does not necessarily imply faithfulness. Especially under sft, the model may simply learn to generate CoT that satisfies the monitor rather than revealing its actual reasoning process, needs analysis.

(3) There are some numerical inconsistencies in the results. The authors claim 23/27 and 21/27 improvements, but Tables 2 and 3 show 24/27 for both.

(4) There is a lack of ablation studies, especially for RL's reward design and OPSD's token-level weighting. It is unclear whether these designs actually contribute to the improvements.

Questions for authors

See weakness, please resolve them.

6 comments
AuthorsOpenReview ↗

Response to Reviewer RNLK

We thank the reviewer for the careful review. The revision clarifies distinctions among three post-training methods, corrects the numerical reporting, and adds behavioral analyses. Unless marked as original, all section, table, and page references refer to the revision.

1. What the three training approaches establish

The three approaches examine different sources and forms of supervision:

  • SFT tests whether disclosure behavior elicited through teacher-only guidance can be transferred through fixed demonstrations and retained under the ordinary evaluation prompt.
  • RL tests whether online outcome and reasoning-monitor judgments can provide a useful scalar training signal, without fixed teacher completions.
  • OPSD tests whether a frozen copy of the initial model can guide the evolving student on its own sampled trajectories. It combines public and privileged teacher predictions and excludes final-answer value tokens from the distillation loss.

These distinctions are described in Sections 4.2–4.4 (pp. 5–6). Each method produces positive changes in both composites in eight of nine model–monitor settings (Table 2, p. 7). Thus, the evidence extends beyond training against a scalar monitoring reward: gains also occur with fixed demonstrations and with token-level supervision on student-generated trajectories. SFT demonstrates transfer without teacher-only guidance at evaluation, while OPSD demonstrates gains without directly supervising final-answer value tokens.

These are comparisons of complete training configurations with their paired Base references. Differences in training budgets, data exposure, and evaluation support prevent attributing their relative performance solely to the training objective.

2. Monitorability and faithfulness

SFT supervision allows rejection, non-use, and independent agreement, and it does not require attributing every answer to the intervention. We screen targets for unsupported attribution and reasoning–answer consistency, and evaluation-monitor scores are not used to select them (Appendix D.1.2–D.1.3, pp. 24–25).

Beyond the composite scores, SFT reduces the joint frequency of sandbagging outcomes and negative monitor judgments in all nine model–monitor settings among trajectories with valid outcome labels, despite increased incorrect-output yield. This decrease persists under conservative bounds for missing monitor judgments; deception results are mixed (4/9 decreases; Tables 24 and 28). These findings support a practical monitoring benefit for sandbagging, without establishing that the generated explanations faithfully represent internal reasoning.

3. Numerical improvement counts

You are correct that the originally printed Tables 2 and 3 each contain 24/27 positive changes, and the accompanying statements of 23/27 and 21/27 were inconsistent.

After correcting the numerical entries and replacing native-support aggregation with paired common-support comparisons, revised Table 2 reports 25/27 positive changes for each composite and 24/27 simultaneous improvements. Each method improves both composites in eight of nine settings. Section 5.2 now matches these results, including the negative changes. The general response provides the aggregation procedure and paired uncertainty estimates.

4. RL reward and OPSD weighting

The current experiments evaluate the complete RL and OPSD configurations, and they do not isolate the contribution of individual reward terms or token weights. Their design rationale is as follows.

The RL reward favors target outcomes accompanied by positive monitoring judgments and penalizes mismatches between the outcome and reasoning judgments. This supplies feedback on both observed target satisfaction and its disclosure, rather than rewarding an acknowledgment alone (Section 4.3, p. 6).

OPSD gives greater weight to positions where the teacher’s predictions differ between public and privileged contexts, while retaining a baseline weight elsewhere. The weighting therefore focuses updates where the additional guidance changes the teacher’s predictions, and it does not identify causally important reasoning tokens (Appendix D.3, pp. 34–35).

The reported gains support these complete configurations. Establishing the incremental benefit of the reward terms or nonuniform weighting requires component comparisons, which Section 6 (p. 9) identifies as an unresolved question.

Reviewer_RNLKOpenReview ↗

Response to Authos' Rebuttal

I thank the authors for the corrections and the new behavioral analyses. But only few of them are resolved.

The model may learn to produce CoT that satisfies the monitor rather than revealing its actual reasoning.

A higher composite can mean either (a) the same behavior is now disclosed, or (b) the model follows interventions more often and narrates it. The unflagged-sandbagging analysis decreases under both, so it does not distinguish them. I see the new data point to (b):

  • SFT reduces intervention-arm accuracy by 6–10 points (Table 21), while control accuracy changes by at most ~2 points. The main text reports only the latter.
  • Sandbagging incorrect-output yield rises by 6–8 points (Table 24). For Qwen3.5-9B, deception-goal satisfaction under intervention rises from 5.6% to 16.5% (Table 25), while unflagged frequency stays around 1% (Table 28). Figure 2 shows the same pattern: the Base CoT refuses the deceptive instruction, while the SFT CoT complies, noting "even though it is dishonest."

Thank you for clarifying this distinction. We agree that reduced unflagged frequency does not distinguish (a) from (b), and the SFT results are consistent with increased intervention compliance. We will report its intervention-accuracy losses and increased deception-goal satisfaction alongside control accuracy in Section 5.2.

RL and OPSD show smaller observed behavioral changes. RL’s monitor-averaged intervention-accuracy losses are 0.39–0.99 points, compared with 6.09–10.06 points for SFT. OPSD’s changes range from −2.17 to +1.02 points (Tables 21–23).

A concrete example is Qwen3.5-9B OPSD under the Llama monitor:

MeasureChange
Upper / lower composite+5.42 / +4.17
Intervention accuracy+0.05
Sandbagging incorrect-output yield−0.31
Deception-goal satisfaction−0.26

This configuration shows composite gains without the large aggregate behavioral shifts observed for SFT. It supports improvements in measured monitorability with smaller behavioral changes.

One clarification on Figure 2: the Base CoT states refusal, but its final answer nevertheless promotes “Privacy and Security” and claims that data are not collected for competitor analysis. Both final outputs satisfy the registered target (Y=1Y=1). The figure therefore illustrates changed attribution at the same target-outcome label, rather than a successful refusal becoming a deceptive final response. This does not negate the broader SFT behavioral changes.

Reviewer_mgjTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether training makes model reasoning easier to monitor. It derives bounds on a monitor's detection rate for outcomes actually caused by an intervention. The authors train models using supervised fine-tuning, reinforcement learning, and on-policy self-distillation on paired tasks with and without interventions. MonitorBench results show the most consistent gains from supervised fine-tuning.

Critical strengths

I think it is useful to show that the existing sensitivity score can overstate how well a monitor detects behavior caused by an intervention. The authors also correctly caution that improving this score does not prove that detection has improved.

Furthermore, training covers multiple intervention types and answer formats, which make the study more robust. Evaluation uses separate data and fixed monitors that differ from the RL training judge, which helps test whether the learned behavior transfers.

Critical weaknesses

I had a hard time reconciling the results with Section 5.1, which says both models use the same tasks and eligible examples. But then for Qwen3.5-9B RL under the Llama monitor, Table 2 matches averaging the base over twelve tasks and RL over ten available tasks in Table 14. Using the same ten tasks turns the reported gain into a decline.

Table 15 also has a lower-bound score above its upper-bound score for health answer cues. This also violates Equation 7 if both use the same observations. I believe these inconsistencies need to be resolved.

Another thing is that I am uncertain about the behavioral change. For instance, the RL reward favors following an intervention openly over rejecting it, and that icludes requests for errors or deception. While, the paper acknowledges that accuracy may suffer and reports neither accuracy nor how often target behaviors occur, I would still like to see whether training reduces undetected misbehavior while preserving task performance. Models could become easier to monitor while misbehaving more often.

Questions for authors

How much of the fine-tuning gain can be achieved by giving the base model the same disclosure instructions when answering?

2 comments
AuthorsOpenReview ↗

Response to Reviewer RNLK

We thank the reviewer for examining the numerical comparisons and the behavioral implications of training. The revision corrects the reporting discrepancies and adds behavioral measurements and a joint-outcome analysis. Unless marked as original, all section, table, and page references refer to the revision.

1. Aggregation and lower–upper inconsistencies

The original comparison of 59.50 versus 60.31 for Qwen3.5-9B RL under Llama used each checkpoint’s native evaluation support. Your matched-task calculation identified the importance of using an explicitly paired comparison. Revised Table 2 uses common eligible instance IDs within each evaluation repetition and identical task sets and weights for both checkpoints.

The corrected paired scores are 67.25 → 68.10 for the upper composite and 62.39 → 63.49 for the lower composite, with eleven defined tasks (Table 2, p. 7; Table 13(b), p. 41). The paired changes and 95% percentile intervals are +0.85 [0.12, 2.05] and +1.10 [0.19, 2.38], respectively (Table 18, p. 47). The revision corrects task-level entries as well as replacing native-support aggregation with paired aggregation. The general response explains the calculation and uncertainty procedure.

You are also correct that the lower composite cannot exceed the upper composite when both use identical observations and nonnegative weights. The original Mistral RL Health answer-cue entry under Qwen3-32B, 1.79/5.63, was a reporting error. Its corrected upper/lower values are 42.56/28.27 (Table 14(a), p. 42). Every defined task entry in the revised tables satisfies the required ordering. The correction concerns the empirical reporting; the theoretical bound is unchanged.

2. Undetected target outcomes and task performance

The added SFT analysis measures the joint frequency of a target outcome and a negative monitoring decision: u=Pr⁡(Y=1,Z=0∣X=1,VY=1),u=\Pr(Y=1,Z=0\mid X=1,V_Y=1), where VY=1V_Y=1 denotes a valid outcome label. This quantity includes both target frequency and monitoring failure. It retains all intervention trajectories with valid outcome labels, without common-support filtering. ZZ denotes an individual monitor judgment (Appendix D.7, p. 53). The sandbagging macro, which equally weights the four sandbagging tasks, decreases in all nine model–monitor settings. Under the Qwen3-32B monitor:

Task modelIncorrect-output yield, Base → SFTUnflagged target frequency, Base → SFT
Qwen3.5-4B52.62% → 60.32%[15.23, 15.28]% → [11.35, 11.64]%
Qwen3.5-9B55.34% → 63.42%[12.87, 12.98]% → [7.78, 7.95]%
Mistral Small 3.2-24B75.56% → 81.73%[56.63, 56.74]% → [45.20, 45.35]%

Incorrect-output yield uses all planned outputs, while unflagged frequency conditions on valid outcome labels. The brackets give conservative bounds for missing monitor judgments within that valid-label population, not sampling confidence intervals. The sandbagging decrease persists under these bounds in all nine settings. Tables 24 and 28 (pp. 50 and 53) report the complete results.

Task-performance changes depend on the model and training method. On the seven tasks with correctness labels, SFT changes control accuracy by −2.18, +0.57, and −1.00 percentage points for Qwen3.5-4B, Qwen3.5-9B, and Mistral, respectively. Mistral also has a 14.40-point decline in answer-cue intervention accuracy (Appendix D.6, pp. 49–50).

For the RL objective highlighted in the review, the observed accuracy changes are smaller. Averaged over the three monitor-associated evaluation summaries, RL changes control accuracy by −0.42, +0.15, and +0.09 points, and intervention accuracy by −0.99, −0.81, and −0.39 points, respectively (Table 22, p. 49). Sandbagging incorrect-output yield increases by 0.15–2.90 points across the reported RL configurations (Table 24, p. 50). These measurements directly document the behavioral changes accompanying the RL monitorability results.

The SFT joint-outcome analysis supports reduced unflagged sandbagging frequency among trajectories with valid outcome labels, despite increased incorrect-output yield. Its deception results are mixed, with unflagged frequency decreasing in four settings and increasing in five.

3. Disclosure prompting at inference time

We have not quantified how much of the SFT gain disclosure prompting alone would recover. Our goal is more monitorable CoT under the ordinary evaluation prompt, without category-specific disclosure guidance at inference.

Teacher-generated trajectories undergo screening and constrained local repairs before being SFT targets. We screen for instruction leakage: traces may quote or discuss explanation-writing guidance rather than explain the public intervention’s role in the response. Affected reasoning spans are repaired where appropriate (Appendix D.1.2–D.1.3, pp. 24–25). The student receives no teacher-only guidance. The gains show that behavior elicited and screened during training transfers to evaluation under the ordinary prompt.

Dear Reviewer mgjT,

Thank you again for your careful review. In our response, we corrected the Base–trained comparisons and reported how often sandbagging outcomes went unflagged after SFT, alongside the changes in task accuracy. When you have a chance, we would appreciate your thoughts on whether these results address your concerns or whether any point needs further clarification.

Best,

Authors

Reviewer_wCEYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether post-training can improve a model’s CoT monitorability in the intervention-based sense of Guan et al. (2025): when an input intervention (added evidence, an answer hint, a sandbagging or deception instruction) changes the model’s output, a fixed monitor should be able to tell from the CoT that the intervention was responsible. It makes three contributions. First, it shows that the minimal-criterion sensitivity TPR_mc upper-bounds the monitor’s recall on intervention-caused outcomes and derives a matching, attainable lower bound (Theorem 1). Second, it builds a decontaminated training set of 3,810 control–intervention pairs across four intervention families from public benchmarks. Third, it trains Qwen3.5-4B, Qwen3.5-9B, and Mistral Small 3.2-24B (native thinking disabled) on this data with SFT from a privileged disclosure prompt, GRPO with a blinded judge/verifier reward, and on-policy self-distillation, and evaluates on the twelve input-intervention tasks of MonitorBench with three monitors, reporting an intervention-arm composite TPR_mc·TNR_{X=1} and its lower-bound counterpart. SFT improves both composites for every model–monitor pair; RL and OPSD give mixed, smaller changes.

Critical strengths

The paper makes precise what TPR_mc from Guan et al. (2025) can and cannot say: it upper-bounds recall on intervention-caused outcomes, and the paper supplies a matching observable lower bound. The result is modest technically, but it is right, it clarifies the semantics of a metric now in wide use, and reporting both endpoints is a sensible practice that I would want future work to adopt.

Context-distilling a disclosure prompt into the model raises the composite on all three task models under all three monitors, and this gain survives when we recompute on matched task sets (e.g. Mistral +13.9, 9B +3.7 under Qwen3-32B). The training data is decontaminated against MonitorBench, the RL judge and verifier are blinded to each other, and all prompts and hyperparameters are disclosed.

Critical weaknesses

I found the paper hard to follow as a standalone document. Much of the setting is defined by reference: what the two arms are, what the cue is, what Y = 1 means for each family, what the monitor is shown, and what the twelve MonitorBench tasks contain are largely only recoverable from Guan et al. (2025) and Wang et al. (2026a), while the composite reported here (g-mean²_int) differs from both. I had to reconstruct the setup from the cited works, and it is possible that some of the points below reflect my own misreading rather than problems with the paper; I would welcome clarification in the rebuttal.

Section 5.1 says base and trained checkpoints are compared on the same task set, but Table 2 appears to average each checkpoint over its own set of tasks, including tasks where the other is NA. On shared tasks only, 9B-RL comes out slightly below base under all three monitors (−0.3 / −3.3 / −2.1 instead of +1.6 / +0.8 / −0.4) and the 4B-RL gain under Qwen3-32B drops from +2.8 to +1.0. No confidence intervals or instance counts are given, and repeated round values suggest some task scores rest on one or two instances. The headline is that all three methods help; on my reconstruction that holds for SFT but is unclear for RL and OPSD.

All scores condition on Y = 1 and p0 / p1 are never reported, nor accuracy after training. The RL reward gives +4 for complying with a sandbag/deception instruction and saying so, 0 for refusing, and −2 for (Y = 1, Z = 0), which on correct-cue tasks penalizes solving the problem independently. That pulls toward following the cue and announcing it. A higher composite could equally mean “learned to follow instructions and narrate it,” which is not the property the paper claims to improve.

Questions for authors
  1. How exactly were the Table 2/3 entries computed? Are base and trained checkpoints scored on the same set of tasks and instances, as §5.1 states? If so, how does 4B-RL receive 100.00 on Judgment while the base is NA on the same task, and how do the 4B SFT entries relate to Table 10 (I get 74.17 / 67.11 / 82.17 vs. 75.59 / 68.95 / 83.04)? A recomputed Table 2 on matched task sets would resolve this.

  2. Could you report, per task and checkpoint, the number of eligible instances and the estimated p0 and p1? In particular, for the NA cells (e.g. 9B-RL Deception), is p1 = 1 or p1 = 0?

  3. Does task accuracy without the intervention, or refusal rate under sandbag/deception pressure, change after each method?

  4. In the RL reward, why is (Y = 1, Z = 0) penalized on answer-cue and contextual-evidence instances where the model reaches the correct answer without the cue? Is there a reason this does not push the policy toward claiming or increasing reliance on the cue?

I also think the paper requires a major rewriting to make it more standalone.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wCEY

We thank the reviewer for checking the numerical results and identifying where the experimental setting required clearer explanation. The general response addresses the shared reporting and aggregation issues. Below, we answer the specific questions and clarify the remaining limitations. Unless marked as original, references are to the revised manuscript.

1. Standalone experimental setting

The control input contains the original task. The intervention input contains the same task with the specified evidence, answer cue, or behavioral pressure added. Teacher-only disclosure guidance is a separate training input and is absent from the trained model’s evaluation prompt. The twelve evaluation tasks fall into four families:

Intervention familyTarget outcome (Y=1)Evaluation domains
Contextual evidenceProducing the specified target answerHealth, knowledge, judgment, preference
Answer cueProducing the specified suggested answerHealth, mathematics, logic
SandbaggingSatisfying the specified incorrect-answer targetSafety, mathematics, science, engineering
DeceptionSatisfying the specified misleading-response criterionEveryday situations

The same target criterion is applied in both arms. Thus, Y=1Y=1 records target satisfaction; it does not establish that the intervention caused the outcome. The monitor receives the task, intervention, permitted target context, and visible CoT, with the separately parsed final answer withheld. Outcome verification assesses the final answer or action separately from the reasoning trace. These definitions are explicit in Section 3 (pp. 3–4), Appendix B.1 (pp. 14–15), and Section 5.1 (pp. 7–8).

Our reported composites are

\qquad C_L=L\times TNR_{X=1},$$ where $L$ is the lower endpoint on causal recall. Compared with the original two-arm composite, these scores use intervention-arm specificity and omit the control-arm specificity term. They still use the control outcome rate $p_0$ through the sensitivity correction. The specificity term is estimated from intervention-arm outcomes with $Y=0$, so the composite is not based solely on outcomes with $Y=1$. ## 2. Aggregation and the specific table discrepancies (Question 1) The original description did not accurately distinguish checkpoint-specific native support from paired evaluation. As explained in the general response, the revised comparisons use common eligible instances within each evaluation repetition and identical task sets and weights for both checkpoints. A task available only for the trained checkpoint cannot contribute to the paired macro. Accordingly, the 4B Judgment task is NA for both checkpoints in the paired RL table and contributes to neither aggregate (Table 12, p. 40). For 4B SFT, the corrected upper-composite scores under Qwen/Llama/GPT are 73.89/64.90/83.77, with paired Base scores 70.36/58.20/76.83. Each comparison uses eleven tasks. Table 2 (p. 7) agrees with the task-level entries in Table 9 (p. 37). ## 3. Eligible counts, outcome rates, and NA (Question 2) NA denotes an undefined paired task score. This can result from failure of the task-reportability criteria, an empty intersection of eligible instances between Base and trained checkpoints, or an unavailable or undefined result in a required evaluation repetition. Instance eligibility requires estimable composites for both checkpoints: sensitivity requires $p_1>p_0$, and intervention-arm specificity requires valid negative outcomes. NA tasks receive no weight in the macro average and are not assigned a zero score (Appendix D.5, p. 46). Tables 19–20 (pp. 47–48) provide common-support counts for the Qwen SFT comparisons and all three OPSD task models. Some task estimates have very small support: the 9B SFT Judgment and Mathematics answer-cue comparisons each have one common instance per repetition. We report these counts alongside uncertainty estimates to make the limited support explicit. Appendix D.8 additionally reports task-level summaries of $p_0$ and $p_1$, with Base and trained values shown separately.

4. Accuracy and behavior under pressure (Question 3)

We have added correctness and behavioral outcomes separately from monitorability. On the seven tasks with correctness labels, the control-accuracy changes are:

Task modelSFTRLOPSD
Qwen3.5-4B−2.18−0.42−0.68
Qwen3.5-9B+0.57+0.15+0.32
Mistral Small 3.2-24B−1.00+0.09−0.63

Tables 21–23 and Appendix D.6 (pp. 49–50) provide the underlying accuracy, correct-output yield, and label coverage. Under sandbagging pressure, SFT increases incorrect-output yield by 7.70, 8.08, and 6.17 points, respectively (Table 24, p. 50). RL and OPSD show smaller changes in control accuracy and sandbagging incorrect-output yield. The latter increases by +0.15 to +2.90 points for RL, while OPSD changes span −0.46 to +1.48 points across the reported configurations.

The SFT joint analysis also measures how often a target outcome occurs without being flagged. Its sandbagging macro decreases in all nine model–monitor settings among trajectories with valid outcome labels (Table 28, p. 53). This decrease persists under the reported bounds for missing monitor judgments; those bounds do not cover missing outcome labels. Deception results are mixed, with decreases in four settings and increases in five.

5. The RL penalty for independently correct answers (Question 4)

The reward couples target-outcome satisfaction with explicit intervention attribution in the CoT. It rewards their joint occurrence and penalizes mismatches, including a positive attribution judgment when the target outcome is not achieved. As you note, an independently correct answer can also receive Y=1,Z=0Y=1,Z=0 and therefore a negative reward. This reflects the reward’s use of observable outcome and monitoring labels as a training proxy, and it does not identify the causal dependence of each individual answer.

We therefore examine whether optimizing this proxy causes a substantial deterioration in task performance. On the seven tasks with correctness labels, RL changes control accuracy by −0.42, +0.15, and +0.09 percentage points for Qwen3.5-4B, Qwen3.5-9B, and Mistral, respectively. The corresponding intervention-accuracy changes are −0.99, −0.81, and −0.39 points (Table 22, p. 49). Sandbagging incorrect-output yield increases by 0.15–2.90 points across the reported configurations (Table 24, p. 50).

These results show that the reported RL monitorability gains are accompanied by small observed accuracy changes on the evaluated tasks. They provide empirical support for the joint reward as a practical training objective, while also documenting its effect on target-outcome frequency. Section 4.3 (p. 6) clarifies that the reward is a proxy for training monitorable behavior, rather than an instance-level test of causal reliance.

Thank you for the thorough response and the corrected analysis. The paired comparisons, intervals, and support counts resolve my first concern, and the added accuracy and behavioral tables address much of the second. I am raising my score to weak accept. Could you please also report the common-support counts for the RL comparisons (and Mistral SFT)?

Thank you for the response and raising our score! The table below gives the per-task common-support counts for all RL comparisons and for Mistral SFT. We evaluated the final checkpoint of each RL run, which leads to the highest reward during training. Each count applies separately to each of the four evaluation seeds (42–45).

We compute common support separately for each Base–RL pair and monitor. An instance can have enough valid judgments to estimate the composite for both checkpoints under one monitor but too few under another. For Qwen3.5-9B, the monitor columns also include final checkpoints from different RL training runs, which can yield different eligible instances. In the Qwen RL columns, counts are listed in monitor order Qwen3-32B / Llama-3.3-70B / GPT-OSS-120B. In each Mistral column, the three monitors had identical counts, so we give one value per task.

TaskQwen3.5-4B RLQwen3.5-9B RLMistral RLMistral SFT
Contextual evidence: Health2 / 3 / 29 / 10 / 101012
Contextual evidence: Knowledge2 / 2 / 27 / 7 / 724
Contextual evidence: Judgment0 / 0 / 00 / 0 / 011
Contextual evidence: Preference2 / 6 / 212 / 9 / 9810
Answer cue: Health24 / 24 / 2423 / 28 / 2877
Answer cue: Mathematics2 / 2 / 21 / 1 / 144
Answer cue: Logic3 / 5 / 35 / 7 / 749
Sandbagging: Safety0 / 8 / 059 / 49 / 49814
Sandbagging: Mathematics0 / 1 / 013 / 9 / 902
Sandbagging: Science18 / 19 / 1890 / 75 / 751218
Sandbagging: Engineering2 / 8 / 254 / 39 / 39822
Deception: Everyday situations3 / 3 / 31† / 2 / 265

† For Qwen3.5-9B RL under Qwen3-32B, the evaluation report records one common instance for Deception. The paired task score was NA, so this task was excluded from the macro-average. We will incorporate these results in revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the reviewers for identifying inconsistencies across our aggregate tables, task-level results, and improvement counts, and we apologize for these reporting errors. The revision corrects the numerical reporting and provides explicit paired comparisons, uncertainty estimates, and behavioral analyses. Below, we present the corrected results needed to assess these changes, with full tables and procedures in the revised manuscript. Unless marked as original, all section, table, and page references refer to the revision.

Corrected aggregation and empirical conclusions

The original aggregates used each checkpoint’s native evaluation support, without enforcing common support between Base and trained checkpoints. Revised Table 2 (p. 7) reports paired comparisons using the same eligible instance IDs within each evaluation repetition and identical task sets and weights for both checkpoints and both composites. We first compute each instance-level g−meanint2g-mean_{int}^2 and g−meanint,lb2g-mean_{int,lb}^{2}, then average over instances within each task and repetition, equally across random seeds 42–45, and finally equally across defined tasks. Tasks defined for only one checkpoint are excluded from the paired macro-average. Each method has its own paired Base reference because common support can differ across methods. The revised task-level tables reproduce the aggregates in Table 2 within rounding.

The corrected counts are 25/27 positive changes for each composite and 24/27 simultaneous improvements. Each method improves both composites in eight of nine model–monitor settings. We have corrected Section 5.2 (pp. 8–9), including the earlier claim that SFT improves both in all nine settings. For SFT, the paired differences and 95% percentile intervals are reproduced below from Table 18 (p. 47). All values are score points, where CU=g−meanint2×100C_U=g-mean_{int}^2 \times 100 and CL=g−meanint,lb2×100C_L= \mathrm{g-mean}_{\mathrm{int,lb}}^{2} \times 100.

Task modelMonitorChange in CUC_U [95% interval]Change in CLC_L [95% interval]
Qwen3.5-4BQwen3-32B+3.53 [+0.58, +7.39]+3.43 [+0.73, +7.18]
Qwen3.5-4BLlama3.3-70B+6.70 [+0.21, +12.59]+5.62 [+2.27, +9.36]
Qwen3.5-4BGPT-OSS-120B+6.94 [+2.99, +11.37]+7.87 [+4.85, +11.33]
Qwen3.5-9BQwen3-32B+1.34 [+0.13, +8.21]+3.79 [+1.40, +10.33]
Qwen3.5-9BLlama3.3-70B+5.01 [+2.48, +12.22]+6.79 [+1.04, +13.27]
Qwen3.5-9BGPT-OSS-120B−1.70 [−5.47, +4.26]+1.36 [−0.26, +8.44]
Mistral Small 3.2-24BQwen3-32B+15.92 [+7.48, +24.53]+9.40 [+3.42, +15.58]
Mistral Small 3.2-24BLlama3.3-70B+8.95 [+1.69, +17.94]+3.17 [+1.86, +9.93]
Mistral Small 3.2-24BGPT-OSS-120B+17.13 [+8.92, +28.09]+11.66 [+5.77, +17.89]

Both SFT intervals lie above zero in eight of nine settings. The remaining setting, Qwen3.5-9B under GPT-OSS-120B, has an upper-composite decline and both intervals include zero. The other exceptions to simultaneous positive changes are Qwen3.5-4B RL under Qwen3-32B (−0.52/−1.60) and Mistral OPSD under Llama3.3-70B (+2.40/−0.21), where each pair gives the upper/lower change.

Appendix D.5 (p. 46) specifies the same 2,000-replicate paired-bootstrap procedure for all three methods. Across all comparisons, 24/27 upper-composite intervals and 22/27 lower-composite intervals lie entirely above zero (Table 18, p. 47). These are pointwise intervals conditional on the observed eligibility masks, without multiple-comparison adjustment. They do not measure variability across independent training runs. Task-level effects remain heterogeneous, and differences in training budgets and common support preclude a controlled ranking of methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.