acceptodds
Withdrawn from ICLR 2027

One Draw Is Not Enough: Resampling-Based Labels for Chain-of-Thought Unfaithfulness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training and evaluating chain-of-thought monitors require reliable labels of whether reasoning reflects what influenced a model’s answer. Cue-based evaluations label a trace unfaithful when the model switches to a suggested answer without crediting the cue, but a single switch can also reflect sampling variability. An existing correction estimates sampling noise in aggregate but cannot validate individual labels. We introduce CueBall, an open-source pipeline that supplements these labels with repeated samples. It measures whether cue-following persists and how each reasoning trace credits the cue. We evaluate five open-weight reasoning models on four multiple-choice datasets with eight cue styles, at temperature 0.7 on questions screened for consistent uncued answers. Among switches from a correct baseline answer to an incorrect cued answer, about 40% of traces labeled unfaithful fail our persistence criterion: the cued answer must recur in at least three of four fresh samples. Across switches to the cued answer, the aggregate noise estimate is closer to the fraction with no observed recurrence than to the fraction failing this threshold. Failure to meet the threshold does not establish sampling noise, and persistence does not establish individual cue causation. We therefore recommend pairing trace-level faithfulness labels with evidence of behavioral persistence. To support this, we release 20,000 traces judged unfaithful or incoherent from 9,997 persistent model–question–cue pairs covering 2,610 questions, with labels, prompts and validation artifacts for monitor and probe research.

Primary area:
datasets and benchmarks
Keywords:
AI Safety, Unfaithfulness, Interpretability, Chain-of-Thought

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Sci8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper's main goal is to improve the measurement of how faithful or unfaithful models are. The authors do this by introducing CueBall, a pipeline that uses resampling to measure if cue-following is persistent or not. In other words, to measure if potentially unfaithful responses are truly so or not. The article's experiments show that 40% of responses labeled as unfaithful are not consistent under resampling.

Critical strengths
  • The strongest contribution is showing that an individual "unfaithful" response might be more attributable to noise than to actual unfaithfulness. The article presents experiments sweeping over five models, four datasets, and eight cue styles.
  • The data for CueBall is open-sourced. This comprises 20k responses that are either truly unfaithful or incoherent.
Critical weaknesses

The main limitation I see for this article is novelty. As the authors note in Related Work, other works in the literature already use re-sampling to estimate the persistence of unfaithfulness. Some are cue-based [1], some are not [2]. The article presented by the authors does strengthen the picture on how persistent or not are the potentially unfaithful responses, but it seems very incremental wrt. previous work.

Besides, there's also little novelty on the CueBal pipeline, which simply collects (resampled) responses and sends them to an LLM Judge. The usefulness of this pipeline is also limited, since it only works on questions for which a cue can be injected.

Finally, there are some clearly AI-generated sentences, but not as bad as I had for other papers. Some examples from abstract:

An existing correction estimates sampling noise in aggregate but cannot validate individual labels. Failure to meet the threshold does not establish sampling noise, and persistence does not establish individual cue causation.

[1]: Chua et al. Are DeepSeek R1 And Other Reasoning Models More Faithful?

[2]: Arcuschin et al. Chain-of-Thought Reasoning In The Wild Is Not Always Faithful

Questions for authors
  • What other interesting insights can we draw from the data collected from the pipeline? Are there scenarios where models are more incoherent? Scenarios where models are more unfaithful? Etc.
  • How can we expand the pipeline to any task that users might use in production?
1 comment
AuthorsOpenReview ↗

Withdrawing while we gather evidence of impact of protocol

Thank you for you detailed review.

We acknowledge the reviewer's concern that the impact of the new protocol has not been demonstrated. We will withdraw this submission while we gather the relevant evidence for impact on probes.

Reviewer_7kR5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces CueBall, a pipeline that supplements cue-based chain-of-thought unfaithfulness labels with repeated sampling. Across five models, four multiple-choice datasets, and eight cue styles, it examines whether initially observed cue-following persists and whether reasoning traces acknowledge the cue. Approximately 40% of positive-case single-sample unfaithful labels fail the proposed 3 of 4 persistence criterion. The paper also provides judge audits, sensitivity analyses, and a dataset of 20,000 traces from persistent model–question–cue pairs.

Critical strengths

The paper provides a substantial empirical study of the stability of cue-based labels across models, datasets, and cue styles. Its transparency is valuable: it explicitly distinguishes recurrence from causal attribution, reports threshold and missing-answer sensitivity, and decomposes differences between protocols into changes in judges, filtering, populations, and weighting. These analyses help readers understand what the results establish.

Critical weaknesses
  1. The paper relies heavily on parenthetical explanations, which interrupt the flow and make the Results section difficult to follow. For example, the statement “positive-case survival at thresholds of one or two target re-rolls is 90.2% or 76.7% (Appendix K)” leaves me unsure what to expect in this appendix.
  2. The importance of resampling is already established in related work: Thought Branches samples both hinted and unhinted conditions, while Monitoring Monitorability explicitly addresses the difficulty of attributing individual outcomes to an intervention. The paper appropriately cites these works, so its contribution rests primarily on the broader empirical analysis and released resource. However, the three-of-four criterion is not calibrated against a substantially larger-sample reference, and the paper does not demonstrate improved monitor or probe performance. Although the authors correctly avoid equating persistence with causation, it remains unclear how much the additional persistence information improves label usefulness relative to its reported 3.8–5.7× token cost.
  3. Section 3.1 reports different top-p/top-k settings for uncued and cued generation, with reroll settings absent from the run metadata. This limits interpretation of uncued–cued differences and reproducibility, although it does not by itself invalidate the descriptive recurrence counts. Separately, the main independent judge audit covers one model–dataset combination; the human golden set contains only 23 traces, with agreement on 17. The separate SSP binary judge and incoherence labels lack independent validation. These limitations matter because the proposed resource is intended to support research requiring reliable labels.
  4. I find the motivation for the four research questions underdeveloped. Although the paper provides empirical answers, it does not sufficiently explain why each question is important or how its answer would improve the training or evaluation of CoT monitors. The connection between the individual analyses and the broader goal of obtaining more reliable unfaithfulness labels needs clearer justification. This makes it difficult to assess the significance of the findings beyond documenting differences between sampling protocols.
Questions for authors

Is there evidence that persistence filtering improves annotation precision or downstream monitor/probe performance, supported by broader human validation?

1 comment
AuthorsOpenReview ↗

Withdrawing while we gather evidence of impact of protocol

Thank you for you detailed review.

We acknowledge the reviewer's concern that the impact of the new protocol has not been demonstrated. We will withdraw this submission while we gather the relevant evidence for impact on probes.

Reviewer_1YpUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines single-sample label instability in cue-based CoT faithfulness evaluations at temperature T=0.7T = 0.7 and introduces CueBall, a resampling protocol (RSP) and dataset. Across 5 open-weight reasoning models, 4 multiple-choice benchmarks, and 8 cue styles, CueBall screens for baseline stability and resamples target switches k=4k = 4 times, defining persistence as ≥3/4\ge 3/4 target recurrence. The paper finds that ≈40%\approx 40\% of positive-case single-sample unfaithful labels fail the ≥3/4\ge 3/4 threshold, show that the aggregate noise estimate 1−α1 - \alpha tracks 0/40/4 non-recurrence rather than <3/4<3/4 non-persistence, and release 20,000 unfaithful/incoherent traces.

Critical strengths
  1. The scale of data collection (141,336 initial cued rollouts plus 8$\times$ baseline and 4$\times$ re-roll completions across 5 reasoning models) and the release of 20,000 persistence-filtered traces with question-level splits provide a useful resource for CoT monitor research.
  2. Comparing 1−α1 - \alpha against 0/40/4 non-recurrence (median gap 1.1 pp) versus <3/4<3/4 non-persistence (median gap 20.1 pp) in Sec. 4.2 clarifies that aggregate distractor normalization removes lucky single-draw flips but retains weak_used pairs (1–2/4 recurrence).
  3. The telescoping decomposition in Appendix K (Tables 15–16) and the non-parametric bounds for missing/truncated answers (Tables 12–13) carefully account for how each protocol stage affects aggregate rates.
Critical weaknesses
  1. There are a few pipeline inconsistencies, such as: (a. Baseline uncued rollouts used top_p = 1.0 without top_k, whereas cued rollouts used top_p = 0.95, top_k = 20, confounding the baseline-to-cued comparison (Sec. 3.1). b). Due to legacy cache omissions, role labels are missing for 78%78\% of Qwen3.5-9B and 29%29\% of Qwen3-8B original switches (Appendix F, K.1), leaving RQ3 incomplete on two of the five models. c). Nemotron re-rolls used a 24,57624,576 token budget while other models used 16,38416,384. Gemma-4-12B had 23.7%23.7\% incomplete re-roll pairs due to truncation; counting truncated re-rolls as off-target misses artificially lowers Gemma-4-12B's reported survival from 90.2%90.2\% (complete-four) to 68.0%68.0\% .
  2. In the post_hoc cue style (Appendix A), the user explicitly instructs the model: "Now walk me through the reasoning that gets to X." A model that derives XX as instructed without stating "I chose X because you told me to" is following user instructions, not exhibiting covert cue unfaithfulness. Yet post_hoc accounts for 45.0%45.0\% (8997/200008997 / 20000) of the released dataset and inflates the pooled RSP unfaithfulness rate from 5.67%5.67\% to 9.13%9.13\% .
  3. The paper motivates RSP labels as necessary for training CoT monitors and white-box activation probes, and notes that recent SSP evaluations use greedy decoding (T=0T = 0) to avoid sampling noise. However, the paper includes no experiments showing that a probe or monitor trained on RSP labels outperforms one trained on T=0.7T = 0.7 SSP (or T=0T = 0 SSP) labels.
Questions for authors
  1. Why were the cached original switches for Qwen3.5-9B (78%78\% missing) and Qwen3-8B (29%29\% missing) not re-run through the GLM-5.3-Flash role judge to complete the dataset before submission?
  2. Does an activation probe or CoT monitor trained on your persistent (robust_used) unfaithful traces achieve higher held-out detection AUROC than one trained on unfiltered SSP traces?
  3. How do the survival rates compare against greedy decoding (T=0T = 0), which is the standard protocol in Chen et al. (2025) and Chua & Evans (2025)?
  • Chen, Yanda, et al. "Reasoning models don't always say what they think." arXiv preprint arXiv:2505.05410 (2025).
  • Chua, James, and Owain Evans. "Are DeepSeek R1 and other reasoning models more faithful?." arXiv preprint arXiv:2501.08156 (2025).
1 comment
AuthorsOpenReview ↗

Withdrawing while we gather evidence for impact

Thank you for you detailed review.

We acknowledge the reviewer's concern that the impact of the new protocol has not been demonstrated. We will withdraw this submission while we gather the relevant evidence for impact on probes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.