acceptodds
Under review as a conference paper at ICLR 2027

CRISP: Validity Criteria for Intervention-Based Chain-of-Thought Faithfulness

Abstract

Chain-of-thought faithfulness is measured by intervening on a model's reasoning and recording the answer change. That contrast is computed under teacher forcing, so it is interpretable only where the replay reproduces the model and the intervention changes what the chain implies. Neither condition is usually tested. We introduce CRISP, a Controlled Reasoning Intervention with a Style-matched Perturbation. One generator writes a semantics-preserving control and a targeted semantic perturbation at matched length and style. The accuracy drop divided by the control's accuracy above chance has reference points at 0 and 1 and needs no baseline. Replay validity requires teacher-forced replay to reproduce the model's own answer, and intervention validity requires the perturbation to lower an evaluator's answer recovery. Both apply to any teacher-forced intervention. Across 15 models and 9 benchmarks, 30 of 135 pairs fail replay validity and 28 more fail intervention validity. Four checkpoints fail replay validity on clinical questions and pass on mathematics. The truncation baseline is negative on 6 of their pairs. Where intervention validity fails, the perturbation changed the recoverable answer on 30% of the checked items per pair against 55% on verified pairs, and where it did, the effect was 0.605 against 0.749. Held-out CRISP is 0.337 higher after CRISP-selected training traces. The criteria add 1 teacher-forced condition and 3 evaluator calls per checked item, so a study can establish that its numbers are interpretable before using them.

Primary area:
interpretability and explainable AI
Keywords:
Chain-of-thought faithfulness, measurement validity, interpretability, large language models, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N1y7OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is mainly concerned with the validity of faithfulness estimates that modify or intervene on models' reasoning traces. Specifically, the paper wants to mitigate the cases where "replay" is invalid. And by "replay" the authors mean cases in which the authors extract the reasoning, append a fixed "Final answer:" cue, and allow a new continuation of up to 1,024 tokens, but instead of producing the final answer, the model just keeps reasoning.

Critical strengths

None

Critical weaknesses

I might be misunderstanding the paper, but it seems the authors are trying to solve a problem that appears just because they did not use proper formatting or chat template? In Appendix D.1, they say:

We did not test whether another cue, such as a native chat template with thinking delimiters, restores consistency on those pairs.

So that means that the failure they observe in models, continuing the reasoning after "Final answer:" cue, is probably attributable to inference setup issues.

Besides that, not all papers in the (un)faithfulness literature use modified reasoning traces, making the usefulness of this approach limited in that regard.

Presentation is also quite bad, and there's a lot of AI-generated content. I struggled to understand what this paper was about. The following is in the first paragraph of intro, which is already citing an Appendix section. What 6 decodes is this referring to? So confusing.

When we perturb 1 clause so that a finding of no association becomes a finding of harm, the model changes its answer from no to yes on every decode. The same edit does not change the answer on any of the 6 decodes for a second chain from that model and benchmark (Appendix E.6).

Also, authors use the expression "reproduce the model" and "replay" all the time, without really explaining what they mean.

Questions for authors
  • Why is "replay" just appending "Final answer:" to the extracted reasoning?
  • Do the experiments use the proper role markers, thinking channel delimiters, etc?
  • What problem is this paper really trying to address?
Reviewer_shFkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CRISP is a protocol for making teacher-forced chain-of-thought faithfulness measurements interpretable. The authors point out that such a measurement can fail silently in three ways: the model does not reproduce its own answer when its chain is replayed, the perturbation does not change what the chain implies, or the perturbed and control chains differ in form rather than content. The protocol has one external LLM write both a paraphrase and a one-error version of the model's own chain at matched length, normalizes the accuracy gap by chance, and adds two pass/fail checks for the first two failure modes.

They run this on 15 open models and 9 benchmarks (math, GPQA, clinical), about 135 model-benchmark pairs. Roughly 40% of pairs fail one of the two checks; the Qwen3.5 family fails almost everywhere on clinical data. Among the pairs that pass, the estimate sits around 0.65 to 0.8 depending on domain. The most interesting experiment is the one that goes against them: swapping the generator for a smaller sibling moves the estimate by a third on average even when both checks still pass. A single-seed finetune on CRISP-filtered traces closes the paper. The authors are candid that no human read any chain and that everything is teacher-forced.

Critical strengths
  • The replay check catches a practical failure that would otherwise be easy to misinterpret. For example, Qwen3.5-9B on MedQA reproduces its own answer on only 4.7% of replayed decodes. Checking this per model–benchmark pair is useful: excluding entire models would also discard 10 replay-valid mathematics/PubMedQA pairs. This is a simple diagnostic that other intervention studies could adopt.

  • The paper reports its sensitivity analyses unusually openly. In particular, the generator-swap experiment exposes a large unresolved source of variation rather than presenting only favorable results. The prompts, decoding settings, alternative controls, and examples also make it possible to examine what the score is responding to.

Critical weaknesses
  • I am not convinced that the experiments distinguish dependence on intermediate reasoning from following the chain’s stated conclusion. The supplied chains can contain explicit answers, and both the tested model and the answer-recovery evaluator can use those answers without relying on the preceding reasoning. Appendix E.6 illustrates the ambiguity: the two examples differ in whether the perturbation also changes the stated conclusion. These examples come from a pair outside the verified set, so they do not establish that the problem persists after verification. But the paper does not provide the corresponding analysis on the verified pairs either. The small average effect of sentence shuffling is consistent with this concern, although it does not settle it. This distinction matters for interpreting a high score as evidence that the model uses the reasoning.

  • Passing both checks still leaves substantial variation in the score unexplained. In Table S12, replacing the 120B generator with its 20B sibling lowers raw CRISP by 0.179–0.500 across six pairs, with a mean decrease of 0.341, although both criteria pass under either generator. On the 50 checked items where both perturbations reduce answer recovery, the difference remains −0.290. The authors acknowledge generator dependence, but I would like a clearer account of what verification establishes beyond screening out two failure modes. The normalized score’s mathematical reference points do not establish calibration across generators. This limits the comparisons that can be drawn from the reported values.

  • Two smaller concerns are the replay setup and the choice of threshold. The appendix attributes most replay failures to non-termination under a fixed answer cue and explicitly limits the finding to this harness. A test using the models’ native formatting would help establish the practical scope of this result. The rationale for choosing exactly ρ₀ = 0.728 also needs clarification: the appendix’s gap concerns model-level clinical minima, while individual pair values fall within that gap and retention changes from 117 to 105 pairs across its endpoints.

Questions for authors
  • On the verified pairs, how often does the perturbation change an explicit answer or conclusion? Could you report the CRISP effect separately for those cases and for edits that leave the stated answer unchanged? A targeted evaluation with explicit answer-bearing statements removed, and the validity checks repeated, would help distinguish use of intermediate reasoning from following a supplied conclusion.

  • Given Table S12, what interpretations or comparisons do you consider justified once a pair passes both criteria? Could you also report control and perturbed accuracies separately under the two generators? This would show whether the difference mainly comes from the perturbations, the paraphrases, or both.

  • How was ρ₀ = 0.728 selected, and was it chosen before inspecting the replay results? How do retention and the main conclusions change at thresholds of 0.6, 0.7, 0.8, and 0.9? If feasible, testing native replay formatting on a few failing clinical pairs would also clarify how much of the observed failure depends on the harness.

Reviewer_QSSJOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper present CRISP, a framework for evaluating chain-of-thought faithfulness using a matched paraphrase and a semantic perturbation. It introduces checks for whether replay reproduce the model’s original answer and whether the perturbation actually changes an evaluator’s answer recovery. Across 15 models and nine benchmarks, 77 of 135 model–benchmark pairs passes both checks. Further experiments looks at generator dependence, surface features, and training on CRISP-selected traces.

Critical strengths

The paper draws a useful distinction between low intervention sensitivity and an evaluation procedure that has failed. Finding 30 replay-invalid pairs and 28 more with ineffective interventions shows why a small accuracy change should not automaticaly be read as unfaithful reasoning (Section 4).

The evaluation spans 15 models and nine benchmarks, with comparisons on shared correctly answered items, bootstrap intervals, and threshold sensitivity analysis. Reporting both filtered and unfiltered results make the consequences of the validity criteria visible.

The generator-swap experiment is especially informative. Switching the generator lowers CRISP by 0.341 on average even though both validity checks pass, which reveals a substantial source of measurement dependence that would otherwise stay hidden (Appendix D.8).

Critical weaknesses

The validity checks dont establish that CRISP isolates dependence on reasoning content. Swapping the generator shifts scores by 0.179–0.500 across six pairs, and a gap of 0.290 still remains on items where both generators produce effective perturbations (Appendix D.8). The authors acknowledge generator dependence, but the remaining variation limit what a “verified” score can tell us about the model itself.

The claim that differences in form are removed “by construction” is not backed by the implementation. Fluency matching is only an auxiliary check, and ten of the 77 verified pairs fail it. The log-perplexity ratio also correlate with CRISP at 0.392 across all pairs (Appendix D.5). Although restricting to matched pairs barely changes the clinical mean, this does not show that individual verified estimates isolate semantic effects.

The prominent replay failures stay tied to a replay setup that is not validated enough. Of 18,682 failed decodes, 18,667 produce no option at all rather then a different answer, and the authors do not test whether native chat templates or thinking delimiters would restore consistency (Appendix D.1). This supports checking replay compatibility, but it leaves the reported failure prevalence and the resulting analysis-set changes depending on an unresolved implementation choice.

Questions for authors

What interpretation is still justified for a verified CRISP score when swapping the generator gives large differences even on jointly effective perturbations? Can intervention strength be calibrated more directly?

Do the main conclusions and individual-pair estimates stay stable when length and fluency matching are enforced as inclusion criteria?

How many replay failures remain when the original generation format, native thinking delimiters, and sufficient answer-generation budgets are preserved?

Could the authors compare CRISP with NSF-CoT (Pramanik et al., 2026) on questions with explicit supporting context to assess whether the two methods identify similar faithfulness failures and which performs bettter? Also using recent and SOTA papers will strengthen the paper

Reference: Pramanik, V., Maliha, M., Bastian, N. D., Velasquez, A., Jha, S., and Jha, S. K. (2026). NSF-CoT: Neuro-Symbolic Formal Verification of Chain-of-Thought Faithfulness in Contextual Question Answering. Findings of ACL 2026, 10645–10663.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final remarks

Dear Reviewers and AC,

Thank you for your time and consideration. If any concern remains open, we would be glad to answer it before the discussion closes on 18th of November (AoE). Tables R1 to R8 are new work for this response. The revised PDF contains every change named below.

Confirmed in the discussion. Reviewer shFk wrote: "The threshold sensitivity analysis, the replay re-run at a larger answer budget, the accuracy breakdown under each generator, and the removal experiment for mathematics have addressed my concerns on those points." Reviewer N1y7 wrote: "The clarification about templates resolves my concerns on the formatting."

1. Replay setup (all reviewers). The replay failures came from an answer budget of 128 tokens. Of the 18,682 failed greedy replays, 18,612 end at this budget. We re-ran 22 of the 30 replay-invalid pairs at 4,096 tokens. All 22 pass replay validity (answer consistency 0.878 to 1.000, Table R1, i.e., Table 1 of the revision). Under the 128-token budget, the truncation baseline is negative on 6 of the failing pairs. A study that took this baseline as faithfulness would have concluded that the models of these 6 pairs ignore their chains.

2. Answer budget and the 77 verified estimates (Reviewer shFk). Some decodes end at the budget before they state an answer. If we count them as correct, the clinical mean changes by 0.023 and the GPQA mean changes from 0.495 to 0.350 (Table R7). On 18 verified pairs decoded again at 4,096 tokens, the clinical mean on the same items differs from the submitted mean by -0.011 [-0.031, 0.008] (Table R8). In mathematics, the estimate from the decodes cut at their 128th token matches the submitted estimate (difference -0.012 [-0.029, 0.003]). Scored on the whole decode, the estimate is lower (difference -0.147 [-0.175, -0.117]), because the models solve the problem again after the answer on their first line. In Section 4 and the caption of Table 2, we now state that the science values depend on the answer budget (Appendix D.16).

3. Stated conclusions in clinical chains (Reviewer shFk). In a test on 16 pairs of 5 models, removing every answer statement from the chains reduced the clinical estimates by 45% and left the mathematics estimates almost unchanged (0.444 to 0.436, Table R5b). After the removal, (M2) passes on 4 of the 8 clinical pairs. In the abstract, the caption of Table 2, and a new paragraph of the Discussion, we now state that a clinical estimate includes the dependence on the stated conclusion. We treat this result as preliminary.

4. Generator dependence (Reviewers QSSJ and shFk). At a matched 128-token budget, the estimates of 6 pairs under the 2 generators differ by 0.299 on average (0.341 as submitted, Table R2). The 6 pairs have almost the same order under both generators (Spearman 0.943). Neither the evaluator's recovery drop nor the change of the stated answer accounts for the difference. In the abstract, we now state that estimates can be compared under one generator but do not give an absolute level of faithfulness. In the Discussion, we name calibration across generators an open problem. Reviewer shFk wrote: "I accept that this is acknowledged as an open problem and I do not expect it to be solved in this paper."

5. Length and fluency matching (Reviewer QSSJ). With matching enforced per item, the estimate of a pair changes by a median of 0.046 under the length criterion and 0.056 under the fluency criterion (Table R3, Appendix D.13). On matched items, the estimates are higher on average. So the residual mismatch does not inflate them on average. We replaced "by construction" with "approximately, through one generator and one instruction".

6. NSF-CoT and recent work (Reviewer QSSJ). On 90 PubMedQA items of 3 verified pairs, we scored the chains with the LLM judge of NSF-CoT. The product of its groundedness and utility scores correlates with the CRISP effect at Spearman 0.336 (p = 0.001, Table R4). The 2 measures agree only in part. We now discuss NSF-CoT in Related Work and in Appendix D.15.

7. Problem, scope, and presentation (Reviewer N1y7). In the first paragraph of the revised introduction, we state the problem and define replay.

Contribution. CRISP tests the 2 assumptions of an intervention-based faithfulness estimate. In this study, replay validity detected the harness failure on 30 pairs. Intervention validity identified 28 more pairs. On these pairs, the perturbation changed the answer recovered by the evaluator on 29% of the checked items with a recoverable answer. On the verified pairs, it changed this answer on 55% (means per pair). At the cost of one teacher-forced condition and 3 evaluator calls per checked item, a study can establish that its numbers are interpretable before using them.

We hope that these results are useful as you finalize your recommendations.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.